1. Field of the Invention
The present invention relates to a data processing apparatus (e.g., a camera signal processing apparatus, etc.) for interpolating a missing color signal into an image signal of each pixel, which is used in cameras having a solid-state image pickup device (e.g., video cameras, digital cameras, etc.). Also, a data processing method using this data processing apparatus, an image processing apparatus using the data processing method, and a camera using this image processing apparatus and a solid-state image pickup device (e.g., video cameras, digital cameras, mobile telephones with a camera, etc.).
2. Description of the Related Art
In conventional cameras (e.g., video cameras, digital cameras, etc., and hereinafter, referred to as cameras), which utilize a single solid-state image pickup device, color filters are discretely disposed on pixels of the solid-state image pickup device.
Each color filter has one of three colors (three primary colors), e.g., red (R), green (G), and blue (B). One of the R, G and B filters is disposed on each pixel. A color (color signal) other than the color of the color filter that is provided to a pixel, can be obtained by calculation. In this case, spatially continuous data (here, two-dimensional) can be obtained as if three color filters were continuously provided. Such a calculation for obtaining spatially continuous data is called interpolation.
Hereinafter, interpolation will be described in detail.
FIG. 8 is a diagram showing an arrangement of colors in a conventional color filter.
Generally, in a color filter plate, G color filters are arranged in a checkerboard pattern, while R and B color filters together are also arranged in a checkerboard pattern where the R or B color filters are arranged on other line (FIG. 8). This arrangement is called a Bayer array.
If a solid-state image pickup device digitizes an image signal via a color filter plate having the Bayer array, each pixel provides an image signal indicating only one color. Therefore, color data, which is not provided to a pixel, needs to be interpolated based on the image signals of the other pixels having the missing color in order to obtain a full-color image signal (including all of R, G, and B colors) for each pixel.
In this case, G pixels, and R and B pixels are interpolated by separate methods.
FIGS. 9A and 9B are diagrams for explaining a conventional interpolation technique. In these figures, the pixels, to which the G filters are provided, are indicated by G. Since the R and B filters are arranged on pixels to be interpolated, the pixels, to which the R and B filters are provided, are indicated by P and Q, respectively.
As shown in FIG. 9A, for example, G33 is assumed to be a pixel to be interpolated. The R and B (P and Q) data of the pixel needs to be obtained by interpolation. The G pixel neighbors are R pixels and B pixels.
P and Q obtained by interpolation of G33 are represented by P33 and Q33. Specifically, P33 and Q33 are calculated based on P's and Q's neighboring G33 by the following expressions:G33=G33P33=(P32+P34)/2Q33=(Q23+Q43)/2.
Next, as shown in FIG. 9B, Q33 is assumed to be a pixel to be interpolated. The G and P data of the pixel is obtained by interpolation. The Q pixel neighbors are G pixels and P pixels.
G and P obtained by interpolation of Q33 are represented by G33 and P33. Specifically, G33 and P33 are calculated based on G's and P's neighboring Q33, respectively, by the following expressions:G33=(G23+G32+G34+G43)/4P33=(P22+P24+P42+P44)/4Q33=Q33.
However, the above-described general interpolation technique raises image distortion and a reduction in resolution at an edge portion, as described below.
Hereinafter, this problem will be described with reference to FIG. 10A, where a vertically striped pattern is imaged.
In FIG. 10A, a line formed of G11 to G15 is a high luminance line, which is typically displayed white. A line formed of Q21 to Q25 is a low luminance line, which is typically displayed black.
FIG. 10B is a diagram showing an image signal value of each pixel in the vertically striped pattern of FIG. 10A, where each white pixel has a value 100, while each black pixel has a value 0.
G33 is now assumed to be a pixel to be interpolated. Interpolation is performed in accordance with the above-described expressions. As a result,G33 =100,P33=(P32+P34)/2=100, andQ33=(Q23+Q43)/2=0.
Similarly, all other pixels are subjected to interpolation in accordance with the above-described expressions. The results are shown in FIGS. 10C to 10E. FIG. 10C shows the values of G's after interpolation. FIG. 10D shows the values of P's after interpolation. FIG. 10E shows the values of Q's after interpolation.
As shown in FIG. 10C, the interpolated data of G's does not represent the vertically striped pattern, thus the image is distorted. As shown in FIG. 10E, the interpolated data of Q's does not represent the vertically striped pattern, where the stripes disappear, i.e., the image is distorted.
An attempt has been made to solve such an image distortion problem caused by interpolation in, for example, Japanese Laid-Open Publication No. 7-59098. In this document, the gray levels of pixels are obtained based on the differences between the horizontally or vertically neighboring pixels. The gray level is compared with a predetermined threshold. Based on the result of comparison, interpolation is performed.
Hereinafter, the interpolation technique disclosed in the above-described document will be described with reference to FIG. 11A, where a horizontally striped pattern is imaged.
In FIG. 11A, lines respectively formed of G11 to G51, G13 to G53, and G15 to G55 each have a low luminance, which are typically displayed black. Lines respectively formed of P12 to P52 and P14 to P54 each have a high luminance, which are typically displayed white.
FIG. 11B shows an image signal value of each pixel of the horizontally striped pattern of FIG. 11A, where each white pixel has a value 100, while each black pixel has a value 0.
According to the above-described document, if pixel Q23 of FIG. 11A is assumed to be a pixel to be interpolated, color data G23 and P23 are calculated by:Q23=Q23P23=0.25×[P14-Adap(G14)+P12-Adap(G12)+P32-Adap(G32)+P34-Adap(G34)]+Adap(G23) G23=Adap(G23)  (1)
Adap in expression (1) is calculated by expression (2) according to the above-described document, which includes conditional branches. Expression (2) below corresponds to pixel G23 of FIG. 11A. In expression (2), “Threshold” represents a predetermined threshold, “Gdiff-hor” represents a horizontal difference in luminance, and “Gdiff-ver” represents a vertical difference in luminance.For Gdiff−hor=|G13−G33|Gdiff−ver=|G22−G24|Threshold=Predetermined value If (Gdiff−hor<Threshold) and (Gdiff−ver<Threshold) or (Gdiff−hor>Threshold) and (Gdiff−ver>Threshold) then |Adap(G23)=(G13+G33+G22+G24)/4|If (Gdiff−hor<Threshold) and (Gdiff−ver>Threshold) then |Adap(G23)=(G13+G33)/2|If (Gdiff−hor>Threshold) and (Gdiff−ver<Threshold) then |Adap(G23)=(G22+G24)/2|  (2).
However, some images may be distorted by the interpolation technique disclosed in the above-described document.
For example, G23 is calculated based on the data values of FIG. 11B in accordance with expressions (1) and (2). As a result,Gdiff−hor=Gdiff−ver=0.
And, irrespective of the value of Threshold,G23=Adap(G23)=|G13+G33+G22+G24|/4=50.
G43 is similarly calculated to be 50.
According to the above-described document, G13, G33 and G53 are all calculated to be 0 based on the data of each pixel.
The resultant data of G13 to G53 is shown in FIG. 11C.
In FIG. 11C, interpolation causes false white signals, which are otherwise black in FIG. 11A, so that a difference in luminance occurs on a horizontal line. The resultant pattern shows alternating black and white or weak and strong luminance, which is called zipper noise. That is, the image is distorted.
Also in the interpolation technique of the above-described document, vertical and horizontal differences in luminance values are compared with the predetermined threshold. This technique is effective if image signals are free from noise and the like. However, if horizontal and vertical differences in luminance value fluctuate near the threshold due to noise or the like, the interpolation calculation fluctuates in a time-series manner, so that data fluctuates. For example, a pixel to be interpolated is likely to flicker in a time-series manner.